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Abstract  -  A  sensate  robotic  gripper  was  developed  and 
interfaced  to  an  electrotactile  tongue  stimulation  system.  The 
prototype  system  permits  grasped  object  recognition  by  the 
user  without  visual  sensory  input.  Modifications  of  an  existing 
two  finger  robotic  gripper  included  the  addition  of  six 
conductive  polymer  force  sensors  mounted  in  a  pentagonal 
(24mm  diameter)  pattern  with  the  sixth  sensor  placed  in  the 
center.  Shape  information  from  the  robot  gripper  in  contact 
with  a  test  object  is  relayed  to  the  user  via  patterned 
electrotactile  stimulation  on  a  micro-fabricated  flexible  tongue 
array.  A  previously  developed  Tongue  Display  Unit  (TDU) 
provides  the  electrotactile  stimulation,  which  pattern  maps 
information  from  the  six  sensors  to  discrete  groupings  of 
electrodes  on  the  12  x  12  matrix  tongue  array.  Modification  of 
an  existing  software  program  facilitated  a  tongue  mapping 
closely  resembling  the  spatial  layout  of  the  six  force  sensors.  A 
preliminary  human  subject  study  was  performed  to 
demonstrate  the  accuracy  of  recognition  when  presented  with 
one  of  four  basic  shapes.  Results  indicate  sensor  resolution  and 
orientation  influence  performance,  but  even  a  limited 
configuration  provides  highly  accurate  shape  recognition. 
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I.  INTRODUCTION 

There  has  been  significant  work  in  developing 
prostheses  controlled  by  people  with  amputations  or  high- 
level  quadriplegia.  Tele-manipulation  systems  were  initially 
developed  for  work  in  hazardous  or  inaccessible 
environments,  e.g.  the  nuclear  waste  processing  facilities 
and  deep-sea  exploration,  where  the  human  operator 
remotely  controls  a  robot  while  viewing  a  video  display  of 
the  end-effector.  However,  even  the  current  systems  are 
slow  and  clumsy,  primarily  due  to  the  lack  of  appropriate 
sensory  feedback  to  the  operator.  The  tele-manipulation 
community  has  long  recognized  the  inadequacy  of  strictly 
visual  feedback,  and  the  particular  need  for  uniquely  haptic 
information  such  as  contact,  grasp  force,  shear,  and  slip, 
which  convey  critical  information  about  the  state  of  the 
hand-object  interaction  [1],  Furthermore,  the  majority  of 
existing  haptic  feedback  systems  (see  e.g.  [2]  for  a  review) 
were  designed  to  provide  feedback  to  the  operator’s  hands 
via  special  displays  or  gloves  in  order  to  stimulate  the 
“normal”  feedback  channels  the  operator  would  utilize  if 
directly  handling  an  object.  However,  these  approaches  do 
not  meet  the  specific  tactile  sensing  and  feedback  needs  for 
a  useful  robotic  prosthesis. 


Models  of  human  manipulation  and  of  fingerpad 
mechanics  indicate  that  the  shape  of  the  finger  deformation, 
distribution  of  force  and  pressure,  and  shape  of  the  contact 
region  facilitates  grasp  stability  and  successful  manipulation 
[3].  For  people  with  quadriplegia,  however,  ordinary  haptic 
feedback  for  tele-operation  is  not  possible  because  both 
sensation  and  motor  control  is  lost  below  the  level  of  the 
spinal  cord  injury.  By  using  the  tongue,  tactile  sensory 
deficiencies  experienced  by  people  with  high-level 
quadriplegia  may  be  overcome  [4,  5].  Employing  this  new 
sensory  feedback  pathway  would  allow  users  to  literally  feel 
the  objects  that  they  are  tele-manipulating.  The  study 
presented  here  utilizes  the  tongue  as  an  alternate  haptic 
channel  by  which  sensory  information  regarding  object 
shape  can  be  relayed. 

Although  this  system  is  intended  for  people  with  high- 
level  quadriplegia,  there  are  clearly  numerous  similar 
applications  for  the  proposed  technology,  or  at  least  the  part 
of  it  that  provides  accurate  tactile  feedback  through  the 
tongue.  Such  a  system  could  be  coupled  to  various  forms  of 
robotic  control,  including  the  use  of  the  hands  to  control  the 
robot.  Operators  with  normal  motor  control  could 
incorporate  hand  feedback  and  tongue  tactile  stimulations  to 
create  an  additional  haptic  channel.  Many  applications 
would  benefit  from  a  staff  of  robotic  hands  sure  enough  to 
entrust  the  handling  of  radioactive,  biohazardous,  or 
explosive  materials.  The  prospect  of  tele-robotic  systems 
that  could  enable  human  operators  (potentially  including 
persons  with  high  quadriplegia  who  become  expert  in  the 
use  of  the  proposed  technology  for  many  life  needs),  sitting 
comfortably  on  Earth,  to  pick  up  and  probe  rocks  on  other 
planets,  or  feel  the  texture  of  object  on  the  ocean  floor,  is 
exciting  to  space  scientists  and  Navy  planners  [6]. 

The  technology  development  studied  here  could 
potentially  be  used  in  such  hands-free  tele-manipulation 
environments.  In  addition,  this  study  is  one  of  the  first 
experiments  conducted  to  demonstrate  the  capacity  of  the 
tele-robot  to  convey  shape  information  in  the  form  of  tongue 
electrotactile  stimulations.  The  experimental  apparatus 
(including  robotic  servo  gripper,  TDU,  and  interface  control 
computer),  in  block  diagram  form,  is  shown  in  Fig.  1, 
below. 
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System  Diagram 


Human  Interface  Robot  Control 


Fig.  1.  Experimental  apparatus  block  diagram  with  outlined  subsystems. 

Robot  Control  and  Human  Interface. 

n.  System  implementation 

Our  overall  goal  was  to  demonstrate  the  feasibility  of  a 
novel  sensate  robotic  gripper  that  could  close  the  control 
loop  in  a  human-telerobot  manipulation  task  by  providing 
tactile  feedback  of  object  shape  information  to  the  users 
tongue.  To  that  end,  we  have  developed  a  prototype  system 
that  affords  a  tactile  interface  between  a  robotic  manipulator 
and  a  human  subject.  The  interaction  exists  at  the  gripper 
face  where  six  force  sensors  (Model  #400,  Interlink 
Electronics,  Inc.,  Camarillo,  CA)  are  attached  and  provide 
shape  contact  information  in  the  form  of  0-5  volt  potential 
changes.  These  voltages  are  converted  to  electrotactile 
stimulations  on  the  user’s  tongue  by  way  of  a  Tongue 
Display  Unit  (TDU-ver.  1.1,  Wicab,  Inc.,  Madison,  WI). 
The  TDU  is  a  programmable  tactile  pattern  generator  with 
tunable  stimulation  parameters  accessed  via  a  standard  RS- 
232C  serial  link  to  a  PC.  The  control  scheme  developed  for 
this  study  is  actually  an  open  loop  structure  because  the  user 
does  not  maintain  control  over  robotic  manipulations.  In 
addition,  visual  and  auditory  clues  about  the  object  being 
grasped  were  prevented  so  that  this  investigation  could 
focus  on  the  capacity  of  the  robot  system  to  accurately 
convey  shape  information  to  the  subject  via  the  tongue. 
Once  this  capability  is  demonstrated,  the  prototype  may  be 
expanded  to  allow  closed  loop  robotic  tele-manipulations. 
The  block  diagram  in  Fig.  1  shows  the  two  interfacing 
subsystems. 

A.  Robot  Control 

For  this  study  we  modified  a  six-degree-of-freedom 
robot  manipulator  (Robix,  Model  RCS-6,  Advanced 
Designs,  Inc.,  Tucson,  AZ).  The  electronic  interface 
provided  with  the  Robix  facilitated  access,  via  computer 
parallel  port,  to  each  of  the  DC  joint  servomotors,  as  well  as 
seven  external  switch  inputs.  Using  five  of  these  inputs  and 
skeletal  software  provided  by  the  manufacturer,  we  created 


a  C++  language  program  that  automated  the  control  of 
robotic  movements.  Once  initiated,  this  program  responded 
to  one  of  five  external  commands.  Four  pushbutton 
commands  were  linked  to  a  unique  set  of  movements,  each 
aligning  the  manipulator  with  one  of  the  predetermined 
object  locations  (labeled  1-4).  The  gripper  then  closes 
around  the  object,  creating  contact  information 
corresponding  to  the  locus  of  the  sensor,  and  waits.  Upon 
depression  of  a  fifth  pushbutton  (labeled  “Return”),  the 
gripper  releases  the  object  and  returns  to  its  “Home” 
position.  The  robot  control  program  was  created  as  a 
graphical  user  interface  (GUI)  and  presented  as  a  basic 
window  containing  a  pull-down  “Options”  menu.  Within  the 
menu  were  three  choices,  “Run,”  “Stop,”  and  “Exit.” 
Initialization  of  the  robot  commenced  upon  depression  of 
the  “Run”  command  and  consequently  enabled  the 
pushbuttons  for  object  manipulation  control. 

B.  Human  Interface 

Realization  of  shape  contact  information  is  provided  by 
six  conductive  polymer  force  sensors  mounted  on  one  face 
of  the  robotic  gripper.  Five  sensors  are  located  in  a  24mm 
diameter  pentagonal  pattern,  with  the  sixth  located  in  the 
center.  Each  7.75  mm  diameter  sensor  has  an  interdigitated 
active  sensing  area  of  5.08  mm,  a  thickness  of  0.38  mm,  and 
30  mm  dual  trace  leads.  Since  the  active  sensing  area  and 
trace  leads  are  of  similar  thickness,  a  ‘force  concentrator’ 
was  added  to  the  active  area  by  applying  a  3  mm  x  3  mm  x 
1.3  mm  (W  x  L  x  H)  square  of  semi -compliant  self-adhesive 
foam  (3M,  St.  Paul,  MN).  This  also  helps  to  reduce  the 
activation  force  dependence  on  sensor  and/or  shape  surface 
height  and  consistency.  To  accommodate  sensor 
dimensional  requirements,  two  16-gauge  aluminum  plates 
(A1  6061,  1@  35  mm  x  32  mm,  1@  35  mm  x  55  mm)  were 
fabricated  and  fastened  to  the  Robix  gripper  using  double¬ 
sided  adhesive  tape  (3M,  St.  Paul,  MN).  Fig.  2a  shows  the 
sensor  attachment  and  spatial  pattern  on  the  aluminum  plate. 


(b) 


Fig.  2.  Thin  film  force  sensors  (Interlink  Electronics,  Inc.),  (a)  Basic  layout 
on  Aluminum  robotic  gripper  plate.  (b|  Sensor  activation  for  triangle  shape. 


Contact  information  from  the  sensors  is  modified  by 
conditioning  circuitry  to  produce  a  0-5  volt  output  to  the  six 
analog  channels  on  the  TDU.  The  stimulation  pattern 
generated  by  the  TDU  is  output  to  a  micro-fabricated 
flexible  electrotactile  tongue  array  consisting  of  144 
electrodes  in  a  12  x  12  matrix.  The  user  may  adjust  the 
relative  stimulation  intensity  with  a  manual  control  knob  to 
allow  for  individual  preference  of  suprathreshold 
stimulation  levels. 

A  final  software  modification  was  made  to  provide 
users  with  a  graphical  feedback.  Using  an  existing  GUI,  an 
image  of  the  robotic  gripper  plate  (see  Fig.  2a)  with  six 
rectangles  representing  the  actual  sensor  pattern  is 
displayed.  Data  from  the  analog  channels  are  digitally 
processed  and  shown  as  a  varying  color  dependent  upon  the 
voltage  magnitude.  As  a  object  is  grasped,  the  graphical 
regions  corresponding  to  those  sensors  in  contact  with  it 
change  from  black  (0  volts)  to  bright  yellow  (5  volts), 
depending  on  a  linear  transform  of  contact  force  magnitude 
(vs),  to  stimulus  intensity  (vj).  The  graphical  representation 
of  what  the  user  should  be  feeling  on  their  tongue  (see  Fig. 
2b)  provides  a  means  of  self-training,  and  affords  error 
checking  of  the  sensor-to-tactile  display  mapping  function. 


III.  Experimental  Configuration 

An  existing  program  was  modified  to  provide  an 
electrode  stimulation  pattern  that  spatially  matched  the 
sensors.  Four  electrodes  were  assigned  to  each  sensor  and 
are  represented  as  gray  areas  in  Fig.  3.  The  labels,  “Al” 
through  “A<5”,  indicate  the  corresponding  analog  input 
channel.  The  stimulation  pattern  on  the  user’s  tongue 
therefore  reflects  the  spatial  information  received  by  the 
TDU  from  the  sensor  array  corresponding  to  the  shape  of 
the  object  grasped  by  the  robot. 

Our  goal  for  the  experiment,  as  previously  stated,  was 
to  develop  a  system  for  the  investigating  the  feasibility  of 
conveying  tele-robotic  shape  information  via  electrotactile 
stimulation  on  the  tongue.  To  facilitate  this  goal,  four 
different  objects  were  chosen:  a  slender  rod  (9  mm  x  46 
mm);  square  (30  mm  x  30  mm);  triangle  (24  mm  base,  45 
mm  height);  and  open  circle  (32  mm  OD,  12.5  mm  ID).  The 
size  and  shape  of  the  objects  were  based  on  sensor 
resolution  and  each  object  provides  unique,  but  not  obvious, 
contact  information.  The  shapes  were  machined  from  6.4 
mm  thick  high  density  polyethylene  (HDPE). 

The  robot  and  objects  were  mounted  on  a  rigid  wooden 
platform  (40  cm  x  45  cm)  to  allow  accurate  and  repeatable 
manipulations.  The  conditioning  circuitry  was  placed  on  a 
stand  above  the  robot,  and  other  components  including  robot 
control  pushbuttons,  Robix  electronic  interface,  TDU,  and 
computer  were  located  adjacent  to  the  platform. 


Back  of  the  tongue 
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Left  Front  of  the  tongue  Right 


Fig.  3.  Tongue  display  unit  electrode  mapping  function  (12x12  matrix). 

Dark  regions  indicate  active  electrodes.  Labels  indicate  TDU  associated 
analog  input. 

IV.  SUBJECT  EVALUATION  AND  RESULTS 

A  human  subject  study  aimed  at  quantifying  the 
accuracy  of  shape  recognition  with  only  tongue  electro¬ 
stimulations  was  conducted  (i.e.  no  visual  feedback).  Five 
adult  subjects  familiar  with  electrotactile  stimulation 
participated  in  this  experiment.  Each  subject  was  first  shown 
the  apparatus  including  the  robot,  the  four  possible  shapes, 
TDU,  and  graphical  display.  Subjects  were  trained  in  TDU 
operation,  and  on  the  sensor-to-electrode  spatial  mapping 
using  the  graphical  display  so  that  they  could  see  and  feel 
each  test  shape  (see  Fig.  2b). 

Once  subjects  could  identify  each  of  the  four  objects 
without  the  aid  of  visual  or  verbal  feedback  when  randomly 
presented,  a  blindfold  was  administered,  and  two  blocks  of 
12  randomized  trials  (equal  representation)  were  performed. 
When  an  object  stimulus  was  presented,  subjects  were  given 
control  of  the  stimulus  intensity,  and  had  unlimited  decision 
time.  Data  was  recorded  as  a  Boolean  1  or  0  indicating 
correct  or  incorrect  shape  identification,  respectively.  If  the 
decision  was  incorrect,  the  “perceived”  shape  was  also 
recorded  for  analysis.  The  overall  results  of  response  from 
the  5  subjects  (120  trials,  total)  are  presented  in  a  confusion 
matrix  (Table  1).  On  average,  subjects  required 
approximately  9.3  minutes  (SD  =  3.1  min.)  to  complete  the 
first  block  of  12  trials,  and  approximately  6.3  minutes  for 
the  second  block  (SD  =  1.1  min.).  Generally,  subjects  did 
not  have  significantly  fewer  errors  in  the  second  block.  This 
phenomenon  promotes  the  idea  that  users  of  a  closed  loop 
electrotactile  tele-manipulation  system  will  learn  to  process 
substitute  sensory  information  to  the  point  where  tasks  are 
perceived  as  unconscious  extensions  of  the  body. 


Table i 

CONFUSION  MATRIX  FOR  OVERALL  SUBJECT  PERCEPTION 
_ (PERCENT  CORRECT) _ 


Actual 

Stimulus 

[Perceived  Shape 
ROD  CIRCLE 

TRIANGLE  SQUARE 

ROD 

0.967 

0 

0.033 

0 

CIRCLE 

0 

0.933 

0.067 

0 

TRIANGLE 

0 

0 

0.967 

0.033 

SQUARE 

0 

0.167 

0.033 

0.80 

V.  DISCUSSION 

The  results  of  our  study  show  that,  overall,  subjects 
were  quite  able  to  correctly  identify  the  four  test  shapes 
using  only  electrotactile  stimulation  on  the  tongue,  despite 
the  low  spatial  resolution  of  the  sensor  array.  In  particular, 
high  recognition  accuracy  was  attained  for  both  the  slender 
rod  and  triangle.  While  somewhat  lower,  the  circle  and 
square  recognition  rates  were  also  very  promising.  Some 
perceptual  difficulty  is  evident  in  five  of  the  120  trials 
wherein  subjects  confused  the  square  for  the  circle  (Table 
1).  Of  the  five  subjects,  one  scored  perfectly  and  another 
identified  only  one  shape  out  of  24  trials  incorrectly. 

Subject  misperception  of  a  square  as  a  circle  is 
interesting  in  that  the  only  difference  between  the  two 
shapes  is  the  presence  of  sensory  information  from  the 
central  force  sensor.  We  speculate  that  this  error  stems  from 
lateral  masking  effects,  where  adjacent  and  surrounding 
sensory  stimulation  inhibits  a  centrally  located  stimulus, 
making  it  more  difficult  to  detect  on  the  tactile  display  [7], 

Another  factor  that  may  contribute  to  the 
misperceptions  is  that  each  sensor  is  fed  through  an 
independent  conditioning  circuit  and,  due  to  variations  in 
electrical  components  and  limited  tuning  capabilities,  the 
voltage  gains  may  be  unequal.  In  addition,  the  force  exerted 
by  the  Robix  gripper  is  not  always  uniformly  distributed. 
Confusion  would  be  introduced  if  object  stimuli  from  sensor 
voltages  were  not  within  a  known  tolerance  of  one  another. 
This  in  fact,  is  a  probable  cause  for  some  of  the  single 
occurrences  of  false  recognition.  Subjects  remarked  that  at 
times  certain  stimulus  areas  on  the  tongue  seemed  weaker 
than  others  and  as  a  result  had  some  difficulty  identifying 
the  object.  A  future  iteration  of  this  experiment  would 
benefit  from  the  use  of  a  rigid  DC  servomotor  where 
parallel  gripper  plates  could  be  maintained  throughout  the 
robot’s  entire  gripping  range.  Subjects  also  reported  the 
existence  of  a  stray  stimulus  associated  with  analog  input 
A2  (see  Fig.  2).  The  location  of  this  errant  electrode  was 
near  the  intersection  of  column  and  row  10.  When  input  A2 
received  sensory  information  (e.g.  the  triangle  shape),  the 
presence  of  a  stimulus  somewhat  near  A3  may  have  caused 
difficulty  for  the  subjects.  Further  analysis  is  scheduled  for 
solving  this  problem. 


VI.  Conclusion 

A  sensate  robotic  gripper  and  tongue  electrotactile 
stimulation  interface  has  been  developed  and  tested.  The 
system  is  intended  to  allow  object  recognition  by  the  user 
without  visual  sensory  input.  Results  from  a  preliminary 
human  subject  study  demonstrated  the  effectiveness  and 
accuracy  of  object  identification  when  presented  with 
multiple  random  instances  of  four  simple  test  shapes  held  by 
a  robotic  gripper.  However,  this  study  is  only  the  first  step 
toward  realizing  a  fully  functional  sensate  gripper  capable 
of  relaying  contact  information  for  complex  objects  and 
orientations.  We  expect  that  future  increases  in  sensor 
resolution  will  enhance  performance,  but  even  the  limited 
configuration  developed  here  provided  highly  accurate 
shape  recognition. 
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